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(54) Enhanced graphical development environment for controlling mixed initiative applications 



(57) The present invention Is directed to a system 
and method which promotes conversational mixed initi- 
ative applications by taking advantage ol additional 
grammars that are active, in one embodiment, tho de- 
velopment environment allows an application designer 
to define a navigational grammar, a universal grammar, 



and a directed dialog call How grammar. The grammars 
can be activo throughout the call How or at selected 
times and are weighted to work with alt of the directed 
grammars. The processing that is followed by each of 
these grammars is defined separately so that each 
grammar can be context sensitive. 
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Description 

CROSS-REFERENCE TO RELATED APPLICATIONS 

[0001] This application is retatod to concurrently filed, 
co-pending, and commonly assigned U.S. Application 
No. 1 0/428,723, filed on May 2. 20D3, entitled "SYSTEM 
AND METHOD TO GRAPHICALLY FACILITATE 
SPEECH ENABLED USER INTERFACES;- commonly 
assigned U.S. Application Serial No. Q9/6B7.296, filed 
October 13, 2000, enlilled "GRAPHICAL PROGRAM- 
MING LANGUAGE FOR REPRESENTATIONS OF 
CONCURRENT OPERATIONS;" and commonly as- 
signed U.S. Application Serial No. 09/603,334, filed 
June 26, 2000, entitled "ENHANCED GRAPHICAL DE- 
VELOPMENT ENVIRONMENT FOR CONTROLLING 
PROGRAM FLOW," the disclosures of which aro hereby 
incorporated herein by reference. 

TECHNICAL FIELD 

[0DO2] This Invention relates to graphical develop- 
ment systems and more particularly to systems and 
methods lor a graphical development environment for 
mixed initiative applications, and even more particularly 
interactive voice response applications. 

BACKGROUND OF THE INVENTION 

[0003] Interactive voice response [I VR) systems have 
been fn use for some lime now. Typically, such systems 
operate such that a calling user is asked a series of 
questions and is prompted to give a response. At first, 
these responses were accomplished by the user touch- 
ing a keypad number. In such systems the calling user 
would be prompted as follows; "Please touch one for 
today's flights and touch two for all ether flight informa- 
tion." 

[0004] More recent system allow the user to answer 
verbally. Thus the user prompts for such systems would 
bo: "Please say one for today's flights and say two for 
all other flight information." An alternate prompt would 
be: "Please say yes if you are flying today and no if you 
are flying any other day." Another alternate prompt 
would be: "Say the time you would like to fly." The user 
would be expected to respond with, "this morning" or 
"five p.m." 

[0005] The desfgner of such systems must code each 
application such that it follows a specific script, or call 
flow. Tools, Including graphical tools using Icons, arc 
typically used for such application can flow coding. One 
example of such a toot is shown in U.S. Patent 
5,946,485, dated August 31, 1999; and U.S. Patent 
6,131, 184, dated Octobor 10 : 2000, both of which are 
Incorporated by reference herein. 
[G006] In such prior an call flow tools, Icons arc used 
to illustrate for the designer the peninem details of the 
call flow so that the designer could rearrange the cad 



flow, or insert other paths or options into the call flow. 
Thus, in the prior art there is a single [con such as icon 
80, shown In FIGURE 8, that a user places in the can 
flow that represents the entire event recognition call 

s Mow. The designer could supply several paramotors that 
are used in defining the particular event to be recog- 
nised. However, in order to change the event recognition 
format, a designer would have to add code to vary the 
structure or process which is to be followed by the stand- 

f° ard ovent handled icon. In the prior art, a single icon 
represents a single recognition event including all ac- 
tions leading to resolution of lhat recognition event. 
[0007] Recently, IVR systems have begun to incorpo- 
rate more complex caller voice recognition events, so 

f s that tho caller might now hear the following prompt: 
•Please tcK mc your flight date and destination city." 
These more complex types or recognition events are 
more difficult to program and to represent by a single 
icon. 

20 [0008] Caller (talker) directed systems rely on the rcc- 
ognition of various responses from the calling user and 
can be as free-flowing as desired. Thus, a prompt could 
be: "Please tell me what I can do for you today.* A more 
typical prompt would be more specific, such as; "Please 

2s toll mo what day you are flying and what flight number 
you are asking about." Speech recognition is then used 
to determine what the talker has responded. 
[0009] The systems discussed above aro called di- 
rected systems in which questions are asked of a called 

30 or calling (talker) user and the system responds to the 
answer to these questions. However, it is desired to al- 
low a user to depart from the scripted questions and ask 
a question of his/her own, or answer a question that has 
not been prompted for by the call flow. These appJica- 

35 tlons are known as mixed initiative applications, and 
present a host of problems for the designer who is trying 
to develop a proper call flow for such a mixed initiative 
application. 

[001 0] One simple example of a mixed initiative appii- 

40 cation would be when the speaker says "help" in re- 
sponse to a directed question. Another example, would 
be when a speaking user, in response to a directed 
question, such as, "Are you ltylng today or tomorrow?" 
answers, "I'm not flying, I have a question." Or answers: 

« "I'm flying next month." Or answers with a question, such 
as: "Do you have late afternoon flights to City XY2?" 
[0011] With the current stale of the art, the application 
designer must anticipate and code each of these sce- 
narios. Current graphical icon application tools do not 

so work for speech recognition applications, and particu- 
larly not for mixed initiative applications. Today in the 
industry, a recognition ovent is handled by defining eve- 
rything inside a singlo icon. So graphically, an icon has 
a bunch of parameters contained within iL The applica- 

55 rjon designer than, in order to customize an application, 
must create code which modifies the operation of the 
exist tng cede associated with the standard icon. 
[0012] The call flow In speech recognition applica- 
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tions relies on the generation, and ultimately the recog- 
nition, of certain grammars. Each grammar is a collec- 
tion of words on phrases that are passed to a system 
component The speech recognition component then 
"listens" to the user input to determine if the user spoke 
one of the defined phrases. If the user spoke one of 
those phrases, that phrase is passed back to the appli- 
cation for subsequent processing within the call flow. 
[001 3J In order lo accommodate mixed initiative ap- 
plications, the designer must have more than one gram- 
mar active at any given time. The first active grammar 
being the grammar associated with the directed ques- 
tlons : i.e the grammar that is "expected" to be received 
from the speaker. The second (or subsequent) gram- 
mars that are active arc those pertaining to a wide range 
of other responses. Some ot these can always be active 
(help, emergency, operator) and others can be active In 
context to the basic grammar (next month, different city 
name. etc). 

[001 4] Also, it is tmporta nt that when the speakor goes 
"oh* script*, the response to that user be appropriate tor 
the words spoken by the speaker. Thus, given the wide 
range or possible responses, the design and subse- 
quent coding or such applications becomes difficult. 
This problem is compounded when a user is in the mid- 
dle of a translation and asks a question so that the user 
can obtain information necessary for continuing the 
transaction. For example, a user might be asked to "say 
the amount you wish to transfer." In response thereto, 
the user asks, "How much money do I have in my sav- 
ings account?". The system has now received an "off' 
script" question and must deal with that answer before 
going forward. 

BRIEF SUMMARY OF THE INVENTION 

[0015] The present invention is directed to a system 
and method which promotes conversational mixed initi- 
ative applications by taking advantage of additional 
grammars that are active. In one embodiment, the de- 
velopment environment allows an application designer 
to define a navigational grammar, a universal grammar, 
and a directed dialog call flow grammar. The grammars 
can bo active throughout the call How or at selected 
times and are weighted to work with all or the directed 
grammars. The processing that Is followed by each of 
those grammars is defined separately so that each 
grammar can be context sensitive. 
[0016] Thus, for example, when a talking user says 
"checking balance." In answer to a question, the system 
can stop what it is doing with rcopoct to that talking cus- 
tomer, and then retrieve the checking balance call flow 
routine, which is then started fresh with that caller. Al- 
ternatively, the system could obtain the checking bal- 
ance and then continue the application it was working 
on with that talking user prior to the user's utterance. 
Thus, the user may be In the middle of a funds transfer 
scenario and may have given the from" account 1 end 



the "xo* account, but for some reason then says: 1 want 
my chocking balance/ Tho system then obtains the 
checking balance using the checking balance call flow 
(or a portion thereof), telling the balance to the user and 

* then continuing with the funds transfer caB flow. At other 
times, and again depending on the context, when the 
calling user says, "I want my checking balance/ the sys- 
tem and method of our invention stops what it is doing 
and enters the full call flow routine for chocking balances 

JO and does not return to what was being done prior to that 
time. 

[0017] The foregoing has outlined rather broadly tho 
features and technical advantages of the present inven- 
tion in order that the detailed description of the invention 

*3 that follows may be better understood. Additional fea- 
tures and advantages of the Invention will be described 
hereinafter which form the subject of the claims of the 
invention. It should be appreciated by those skilled in 
the art that the conception and specific embodiment dis- 

?c closed may be readily milled as a basis for modifying 
or designing othor structures for carrying out the same 
purposes of the present invention. It should also be re* 
alized by those skilled in the art that such equivalent 
constructions do not depart from the spirit and scope of 

25 the Invention as set forth In the appended claims. The 
novel features which are believed to be charactortstic of 
the invention, both as to its organisation and rnothod of 
operation, together with further objects and advantages 
will be better understood from the following description 

so when considered in connection with the accompanying 
figures. It is to bo expressly understood, however, that 
each ol the flguros is provided for tho purpose of illus- 
tration and description only and is not intended as a def- 
inition of the limits of the present Invention. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0018] For a more complete understanding of the 
present invention, reference is now made to the follow- 
40 ing descriptions taken in conjunction with the accompa- 
nying drawing, in which: 

[0019] FIGURE 1 is one embodiment of an anticipat- 
ed mixed use call Mow graphical representation; 
[0020] FIGURE 2 shows a single slot graphical dls- 
45 play of a *no match", 'no Input 1 loop; 

[0021] FIGURE 3 shows a multi-slot graphical display 
of match conditions; 

[0022] FIGURE 4 shows the validation loop; 
[0023] FIGURES 5. 6, and 7show examples of graph- 
so ical forms used to establish and control eat) flows; 
[0024] FIGURE d shows one embodiment of a navi- 
gational grammar; 

[0025] FIGURE 9 shows examples of inline behaviors 
for universal grammars; and 
99 [0026] FIGURE 10 shows an example of message 
control for delivering messages in context 



45 



so 
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DETAILED DESCRIPTION OF THE INVENTION 

[0027] In the discussion that is to follow the term 
grammar is used to mean a phrase, or collection of 
phrases, that is stored in anticipation of a calling (or 
called) user unering one of those phases in response to 
a question presented to the calling (or caned) user 
Thus, if the presented question is "What is your favorite 
color?", the 'grammar' would con$i$t of a list of colors. 
In response to the question: "What is your favorite 
color?" the user could name a color. The speech con- 
taining the color utterance would then be passed to the 
component that matches the received utterance (gram- 
mar) with the prestored grammar. If the answer matches 
any of the items in the prestored grammar then a valid 
indication is passed back to the application together with 
the color response in coded form. Also passed back to 
the system, Is a code representing the level of confi- 
dence the system has in knowing which color whs ul- 
teied. II the answer does not match trie presented ques- 
tion (i.e.. the talker said "hot"), or if the confidence lovel 
Is below an acceptable threshold, then a mismatch con- 
dition occurs. The mismatch could be because the talker 
answered by naming a color not on the list, or the talker 
could have spoken a word or phase out of context, Or a 
no-lnpur condition could have occurred if the talker did 
not speak at ail. 

[0028] Turning now to FIGURE 1 , call flow 1 0 is a di- 
rected dialog call flow and can be controlled, for exam- 
ple, as shown in the above-identified copending appli- 
cation. Thus, in the funds transfer application, there 
would bo a function caltod funds transfer 101. This Is a 
sub-routine actually where the system says, via 101, 
*Please say the amount and accounts for the funds 
transfer.'' In that context , il the talker wants to take over 
the call flow and request his/her checking balance, this 
can be accomplished because the system has defined 
'checking balance* to be a navigational grammar, as 
shown on asynchronous screen 11 . 
[0029] The navigational grammars are defined on the 
asynchonrous screen so that the user can interrupt the 
directed dialog call How at any time. The processing for 
what the caller wants will bo accomplished under the 
call now for the asynchronous page. 
[0030] In operation, in one embodiment, there are two 
pages, one for asynchronous events (screen 11) and 
one for directed answer events (10). When a user is 
working under call flow 10 and says an out-of-context 
word or phrase, as controlled by Icon 106. the system 
remembers the context In which such word Is said and 
then processes the request via the asynchronous page. 
Because the system is in the middle of a funds transfer 
and the programmer has established that the context 
Should be remembered if a non-script event occurs, the 
system will coma back and complete the funds transfer 
routine. 

[0031] Altar retrieving the proper balance, the speak- 
er would then have his/her account balance so the 



speaker would then be ready to proceed with the appli- 
cation he/she was using prior to the 'interruption, 4 Some- 
times the speaker will ask for information necessary for 
the active routine and when that information is retrieved 

5 it Is plugged directly Into the routine. For example, as- 
sume a caller is trying to report an auto accident and 
calls his/her insurance company. The IVR answering 
system might ask for the policy number. The user could 
respond, "what is my policy number?**, the system could 

to then launch into a routine lor asking questions to identify 
the caller so as to retrieve his/hcrpolicy. When the policy 
number Is retrieved, Hcan be spoken to the caller, or for 
privacy, it might just bo entered Into the original accident 
reporting routine and that routine restarted where it left 
off. Perhaps the callerwouldbemfd, "We have identified 
your policy. Now toll us where the accident occurred." 
[0032] The navigation page, as controlled by icon 
106, Is a page where phrases are Identified that basi- 
cally change the context. In the example being dis- 

£0 cussed, where the user asked lor his/her account bal- 
ance, the system understood that the context is being 
changed. Accordingly, the first thing that must happen 
is a recognition that the context is being changed. 
[0033] Let us assume that the talking user is in the 

25 process of listening to rato information and following a 
menu of presented options and the user says, "What is 
my mortgage payment?". This is a navigation phrase or 
a change in context phrase. However, because the con- 
text of the call flow is a rate application and the question 

30 is one thatwas anticipated forrate application call flows, 
the system will answer the payment question and then 
como back to the rato application. The system remem- 
bers the context of the current conversation, but still an- 
swers the caller's question. 

as [0034] The operation of the system and method of 
FIGURE 1 will now be discussed with reference to FIG- 
URES 2, 3 and 4. 

[0035] Turning now to FIGURE 2, icon 21 shows what 
the prompt would be for one branch, branch 20, of the 

40 call flow. FIGURE 2 shows primarily the no match, no 
input portions of the call flow in a graphical manner. 
[0036] Icon 21 Is a user prompt asking the user to 
state the desired account number two digits at a time. 
The spoken response from the talker is passed to voice 
recognition components (not shown} over path 202. At 
this point. Tor this presented question, the grammar has 
been preset, and consists of numbers. 
[0037] The system graphically displays the process- 
ing of the call flow when a word outside of the grammar 

so Is spoken, which is the no-match situation. The no- 
match display shows two possibilities; a mismatch 22 
and a no-input 23. As discussed above, when the talker 
responds with a word outside of the grammar, as shown 
by icon 22, there is a mismatch condition. When no input 
Is detected, as shown by Icon 23, there Is a no-input 
condition. In each of these conditions, tho designer can 
define the subroutines 2402, 2502 that will be followed 
by simply moving the icons to their appropriate position. 
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[0038] Path 201 Is a looping construct- Tho dlsptay 
shows lines that represent the bohaviorof looping which 
is back around to prompting icon21 . The designer using 
the system and method discussed, may. If desired, 
break the loop. This break could result In a call going to 
an operator, or a simple hang-up, or by a message being 
played to the caller telling the caller that the session is 
being terminated. 

[0039] Screen 24 controls the no match and no input 
situations for exceptions, I.e. when a user says a word 
or phrase out of context. Icons 25, 26, 27, 28, 2601 and 
2602 control the functions as discussed above. 
[0040J Turning now to FIGURE 3 there Is shown call 
flow display 30 which fe an example of an exception rou- 
tine where the talking user cannot find his/her account 
number. The call flow tries to help, via icons 303-1 by 
changing the message to the user, trying to help me user 
by asking (or telling) different questions or Instructions. 
When «Q else fails, screen 31 and icons 310,31 3-1 con- 
trol the transfer (or hang up) functions. 
[0041] FIGURE 4 shows call flow 40 which Illustrates 
the validation process. Call flow 40 Is essentially the 
same as call flow 10 (FIGURE 1). oxcopt that a leg has 
been addod starting with icon 41 and including icons 42, 
4a, and 44, Call flow 40 follows tho form shown In FIG- 
URE!. 

[0042) Within teg 41 f a slot is defined that is never 
filled by the talker since it actually is used only when the 
legs are alt filled. Icon 41 will repeat to the caller what 
the caller has requested, by saying, "Do you want to 
transfer $500 from account XXXX to account yyyy?*. 
Icons 42 and 43 verify thai the caller has answered with 
the proper grammar, such as "yes 1 *, •okay," or an equrv- 
alont affirmative response. If the transfer is validated, 
Icon 44 controls tho flow of information to the proper lo- 
cation so as to control the transfer. If the transaction is 
not verified, then the system loops back and restarts. 
[0043] FIGURES 5 and 6 show one graphical repre- 
sentation of the Implementation of FIGURES 1-4. FIG- 
URES, shows form 50 which deals with single fields and 
allows a user to Indicate an initial prompt, such as "Say 
your account number,- 51 . Tho variable 52 is Identified 
that wilt hold the recognition results. Field 53 allows the 
user to pick a grammar that Is going to be active for the 
recognition instance defined by the form, or, If desired, 
the user can write his/her own grammars). There is a 
list on the form to support customizatlons and there can 
be as many grammars active as desired for this recog- 
nition instance. In the example, "number* grammar 503 
is selected. 

[0044] Form 50 also provides fiold 54 to hold the detail 
recognition results when necessary. The detail recogni- 
tion results allow tho userto obtain as much information 
as possible about the result For example, the confi- 
dence scoresfor a particular recognition event and other 
details arc available to aPow the user specific proccss- 
Inrj. Thus, for any particular recognition event, the user 
can tako the high road and trust the recognizor to pro- 



vide the right piece of Information or the user can take 
the detail road and base his/her next action on the level 
of detail (for example, confidence tcvef). 
[0045] Irt addition, form 50 wilt allow specific proper- 
3 ties 55 to be passed to the recognizer. For example, 
when an account number Is being requested, the rec- 
ognition engine could be told the length of tho number 
to expect. 

[0046] Events field 56 is used to control selected 

to events, such as. for example, lime out. The user can 
determine the type of processing desired when a time 
out occurs. Also, tho user can invoke a global definition, 
(time period), or can provide a specilic period for this 
instanco of recognition. 

f5 [0047] Sometimes a responding user (talker) says 
something that does not fit Into the grammar. This is a 
no-match condition. The programming user might want 
to do something specific for a no match, such as provid- 
ing a prompt with an example in it. Tho programming 

20 user might want to continue the answer. For example, If 
the recognizer believes it understood what was said, but 
is only 50% certain, the application can then play back 
the perceived value and then "listening" for a *ye& or'no/ 
[0048] Anoihor type of property is confirmation 
thresholds themselves. What percentage of certainty b 
going to cause the confirmed lag to be executed as a 
recognition? Thus, a programmer can at every instanc- 
es of speech recognition Indicate what percent certainty 
will be tolerated at that point. For example, It is harder 

30 to recognize 123ABC, than it Is to recognize Sally, a 
prompt name. The programmer might raise the confir- 
mation thresholds when recognizing certain types of 
grammars. That is a benefit of having these properties 
and having them ba instanco related. 

05 [0049] FIGURE 6 shows from 60 which is used for 
multiple lields which are essentially complex prompts 
where the application is expecting multiple pieces of In- 
formation to come back from tho talker This Is shown 
in FIGURES 1-4 graphically. Form 60 gives the pro- 

40 grarnmer the abflity to define the initial prompt 61 and to 
define the variable 62 that is going to hold the phrase. 
This associates the phrase result with the slot value 63 
Inside a grammar beca use th ere Is o ne gram ma/ file that 
understands the entire phrase that is expected. When 

45 the event recognizer understands multiple pieces of in- 
formation, it passes back that information to the appli- 
cation, using slots. Column 64 is called Initial Value and 
is used when the application already knows a particular 
value. Column 62 is empty on tho form but let's assume 

so a variable is put there called "My Money." When tho ap- 
plication Is being executed and My Money has a value, 
then the system need not try to interpret what is said, 
and need not even ask the question. This is a program- 
ming construct that simplifies the amount of code need- 

55 ed. If the field is populated with a specific valu c instead 
of having the system say, "speak the amount for funds 
transfer," the system begins by saying, "What account 
would you like to transfer $XX dollars from?* 
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[0050] RGURE 7 shows confirmation form 70 which 
controls as validation Icons 41 and 43 (FIGURE 4). The 
tegs on it arc Yes, No and Unknown. Form 70 contains 
paramotors that define the time out value 71. and the 
units for the time out 72, and the results 73, which is the 
string of Yes. No, or an Unknown value. The reason for 
using a confirmation form is that the form forces the au- 
tomatic changing of the grammar for further processing. 
For example, if an answer has been received and the 
question asked in rotum is, "Did you say, 1 , 2, 3, 4, S?\ 
Then, if the talker says no. the system knows that 1 , 2, 
3, 4. 5 is an invalid answer and those numbers (or words 
or phrases) are removed form the grammar for the next 
try, So what it means Is that by using this form, the next 
recognition event should have improved results. This 
process can be applied, for example, to a 'name' gram- 
mar Assume the grammar contains a bunch of names, 
including Mabel, Mildred, Manny. Assume also that in 
response to the question, n/Vno are you calling?", the 
talker says, "mumble, mumble." The system then says. 
"Did you say Mabel?*. Tho talker's response is TMo.-Thc 
system could thon remove Mabel from the grammar and 
ask the talker to repeat the name. 
[0051] The selection of grammars, or the elimination 
of certain words from a grammar, can bo very helpful. 
For example, assuming nam© recognition and assum- 
ing a male name grammar, a tomato name grammarand 
a generic name grammar. In response to a received 
name the recognizer returns a low confidence score, but 
reports thai It Is more confident that It was a femaio 
name and not a male name. 

[0052] In this situation, the caller can be asked to re- 
peat the name, but this time the male and generic gram- 
mars are off. Also, it could bo that the name with the 
highest confidence score Is Sam, but the recognizer in- 
dicate* that it has a high confidence (perhaps based on 
other characteristics or other information) that the 
speaker name is a female name. Then the system would 
accept the female name with the highest confidence 
score, even though a male name (Sam) had a higher 
absolute score. The male-femaJe confidence level could 
come from many sources, including a database of ex- 
pected results dependant on other known results, or the 
speech itself could yield clues. Examples of such clues 
could be a speech pattern or power, or frequency com- 
ponents of the speech. 

[0053] FIGURE a shows one embodiment of the nav- 
igation grammar. The navigation grammar shown In 
FIGURE $ Is whoro iho programmer graphically Indi- 
cates (S1-B3) what short cut phrases to be on the look- 
out for. These could be checking balance 81 , savings 
balance B2, money market balance 83, or any other an- 
ticipated phrases pertaining to the session being con- 
ductod. Those shortcut (or 'Jump*) phrases are graphi- 
cally shown and can be added. Icon by icon, at any time. 
These phrases can be activated any time during the call 
flow. Once the system 'hears' ons of these phrases, ft 
jumps to the propcrcall flow routine to obtain the desired 



information, as shown in- FIGURE 1 . 
[0054] Two scenarios must be looked at First, the 
talker couid actively be desiring to go to a new call flow 
application or the talker could be looking for information 

5 in order to complete the current call flow. This determi- 
nation is made by establishing 'anticipating' phrases for 
each application. If the talker says any of these antici- 
pating phrases, the system knows to hold the place and 
retrieve the desired information. If the talker says an out* 

i0 of-contoxl phrase not in the 'received' list, then the sys- 
tem begins a new call flow routine, based on the word 
or phrase received. 

[0055] Form 85 on FIGURE 8 shows some of the an- 
ticipated phrases for a funds transfer call flow and the 

% * programmer can add as many as desired, which will 
show up on the icon graphical display shown on tho left. 
This is a mixed initiative where the talker has control 
over the conversation but tho system remembers the 
eonlexL because the *neW conloxl is a related context 

so on the icon graph that is shown as being inlined. 

[0056] One method of adding inlined call flows, such 
as checking balance, savings balance, etc. Is to double 
click on plus sign 001 boated underneath icon 84. This 
odds a new leg identical to ono of the speech navigation 

2* link logs, basically indicating that this is a replacement 
for the generic processing defined on that async page. 
With respect to FIGURE 1 everything that is bolow plus 
icon 801 is considered optional, while everything above 
Is mandatory. This allows for a vary high level picture for 

30 a significant amount of can flow that is being handled for 
a caller, 

[0057] FIGURE 9 shows screen 90 for controlling In- 
line behaviors for universal grammars. This environ- 
ment allows a programmer to identify the grammar to 

& contain universal (available throughout the call flow) 
words or phrases as well as the action to take when one 
of those phrases is spoken. Screen 91 alJows for the 
definition of the universal grammar and provides a de- 
fault weight for phrases contained in the grammar. A uni~ 

-*o versal grammar is a set of words, such as Help (904), 
Exit (905), Cancel (906). Transfer to Operator, etc. Upon 
detection of any phrase in the global grammar, the sys- 
tem would normally just switch from the current call flow 
to the call flow associated with the received command. 

43 However, the environment of the disclosure recognizes 
that there might be somo Inllnlng or Instant specific be- 
haviors that the programmer might want to take advan- 
tage of with regard to these universal phrases, as dis- 
cussed above. 

90 [0058] FIGURE 10 has the phrase Help (1001) in- 
lined. Help Is defined on the universale page, but since 
it is inlined in this call flow thorn will bo context specific 
behaviors for this recognition ovont. For example, if the 
quostion presented to the talker is, "What Is your ae- 

ss count number?* (as shown by icon 1100) and the talker 
says. "Help!*, instead of transferring tho talker to the 
general help call flow, the response might be; "Your ac- 
count number Is located on tho bottom right comer of 
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your check", or. "Your account number can be seen in 
the top leftconwof your last statement. Because 'Help* 
has been added as an Inline in the devolopmont envi- 
ronment, not only can the system provide instance in- 
formation it can also provide logic around that instance 
information. Also, every time help is asked for. a different 
message can be used. Thus, as shown in FIGURE 10, 
Help message 1 002 Is played the first lime, while mes- 
sage 1003 is played the second time, and message 
1004 is played the third time. 

[0059] Another bcnolil of having the inline associa- 
tion, is that conversational prompts are possible. Thus, 
a whole call flow could be created to' say okay, get a 
check in front of you and look at the bottom right comer. 
Or, the flow could ask if the talker has his/her last state- 
ment If so, look at the bottom right comer. It the talker 
says 'no*, then another can new could be used to try to 
obtain the account number This is an available because 
the programmer can add legs, simply by adding or mov- 
ing Icons around on a graphical layout. 
[0060] Although the present invention and Its advan- 
tages have been described in detail, it should bo under- 
stood that various changes, substitutions and altera- 
tions con bo made herein without departing from the 
spirit and scope of the invention as defined by the ap- 
pended claims. Moreover, the scope of the present ap- 
plication is not intended to be limited to the particular 
embodiments of the process, machine, manufacture, 
composition of matter, means, methods and steps dc- 
scribed in the specification. As one of ordinary skat in 
the art wilt readily appreciate from the disclosure of the 
present invention, processes, machines, manufacture, 
compositions of matter, means, methods, or steps, pres- 
ently existing or later to be developed that perform sub- 
stantially the same function or achieve substantially the 
same result as the corresponding embodiments de- 
scribed herein may be utilized according to the present 
Invention. Accordingly, the appended claims are intend- 
ed to include within their scope such processes, ma- 
chines, manufacture, compositions of matter, means, 
methods, or steps. 



Claims 

1 . A call flow routine comprising: 

a directed call flow routine established by one 
or icons, each icon representing a leg or said 
call flow, each of said legs of said directed call 
flow necessarily populated in order to complete 
said Directed call flow routine; and 
an optional call flow routine established by 
icons, each icon ropresentlng a leg of said op- 
tional call flow routine, each said leg optionally 
Invoked under control of a detected phrase out- 
side of the phrases expected by said directed 
call flow. 



2. The calt flow of claim 1 further comprising: 

returning to said directed call flow after com- 
pleting an invoked optional call flow when said 
5 detected phrase is from an anticipated gram- 

mar associated with said directed call flow. 

3. The call flow of claim 2 further comprising: 

to beginning an entirely new call flow when said 

detected phrase Is not from an anticipated 
grammar associated with said directed call 
flow. 

is 4. The call How or daim further comprising: 

changing t ho message of said optional call now 
depending upon the context In which said op- 
tional call flow was invoked. 



20 



5. The call flow of claim 1 further comprising: 

changing the message of said optional call flow 
depending upon the context In which said op- 
tional call flow was Invoked and the number of 
times said specific optional call flow has been 
invoked within a specific session. 

6. A graphical toot for use In interactive response sys- 
tems, said tool oporabte for displaying to a call flow 
designer the logical operation of an application: said 
tool comprising: 

means for transmitting prompts to a talking us- 
er, said prompts including Instructions for ver- 
balizing a response in at least one word chosen 
by the talking user 

means for receiving from said talking user an- 
swers to said prompts, said answers including 
at least one word spoken by said talking user; 
means for returning to an application matches 
between what a talking user has responded 
and a preestabHsbCd grammar: and 
means, including the placement of Icons, lor es- 
tablishing and displaying a call flow routine hav- 
ing at least one slot return, said slot return Iden- 
tifying at least one phrase outside of the expect- 
ed range of responses. 



so 7. The tool of claim 6 further comprising: 



means, Inducing the placement of icons, for 
identifying returned ones of said phrases whfch 
are outside the expected responses as being 
within an anticipated jump phraso. 
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8. The tool of claim 7 further comprising: 
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means for fulfilling a talking user's request 
based upon a received phrase outside the ox- 
pected responses: 

means for returning said talking user to said 
prompts when said request has been fulfilled 
and said request was within said anticipated 
jump phrasos; and 

whorcin said fulfilling means is part of the cad 
flow established and displayed by said icon 
placement means. 

9. A eomputersystem having a display, said computer 
system providing a graphical development environ* 
mcnt for developers, said computer system com- 
prising: * 

means ror graphically representing an applica- 
tion flow as a plurality of icons, some of said 
Icons representing sub-routines; 
means for manipulating said icons, including 
said sub-routine icons, into single slot and mul- 
tiple slot returns, each said return handling a 
different expected functional response from a 
talking user; and 

moans for creating a loop for controlling out of 
context verbal responses according to the na- 
ture of each said response. 

10. The computer system of claim 9 wherein said con- 
trolling means include the pre establishment of 
phrases in context to said application flow. 

11. The eomputersystem of claim 10 wherein said con* 
trolling means further comprising: 

means for returning to said application flow af- 
ter digressing to another application, said re- 
turning moans controlled, at least in pan:, by de- 
tection of one of said proostaMshed phases. 

12* The method for programming an Interactive re- 
sponse system where at least some of the interac- 
tive responses include bidirectional voice prompts, 
said method comprising tne step of: 

arranging Icons to form a plurality of return 
slots, each return slot having subroutines 
shown by other icons, said subroutines estab- 
lishing conditions for application operation; 
at least some of said return slots are operabfo 
lo receive returns In response to a grammar 
against which returned voice prompts from a 
talking user is matched; and 
at least some of said roturn slots arc operable 
to recoive returns in response to a grammar 
outside of the oxpected grammar but within a 
related context to said oxpected grammar. 



13. The method of dalm 12 further including the step of: 

arranging said icons such that when a non-ex- 
pected grammar Is received, next system opcr- 
5 ations arc dependent upon whether or not said. 

unexpected grammar is within said related con- 
text grammars. 

14. A method for controlling IVRcall flows; said method 
10 comprising: 

establishing a plurality of grammars, each 
grammar having contained therein expected 
responses to a given question; 
15 using icon controlled proostabltshed routines 

for establishing, for a given compound ques- 
tion, which grammars are to be used for each 
answered portion of such compound question; 
and 

20 using icon controlled p reestablished routines 

tor controlling subsequent call flows In re- 
sponse to an unexpected answer to a given 
question. 

2S 15. The method of claim 14 further including: 

when an unexpected answer has been re- 
ceived, performing the desired operation con- 
tained in said proestablished routine. 
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16. The method of claim 15 further including: 

after performing said desired operation, taking 
further action dependent upon the context of 
said unexpected answer. 

17. The method of claim 1 5 further Including: 

after performing said desired operation, taking 
further action dependent upon whother or not 
said unexpected answer has boon prctdentilied 
as being in context to said given question. 

18. A graphically controlled IVR system, said system 
comprising: 

a system for establishing a call flow having mul- 
tiple slots for receiving verbal responses to a 
compound question, each slot associated with 
a grammar appropriate for an expected re- 
sponse; 

circuitry for directing each such verbal re- 
sponse to a grammar for Interpretation in ac- 
cordance with an expected grammar for said 
response; 

circuitry for receiving in an appropriate slot, in- 
terpretations from said grarnmarwhen a verbal 
response is proper; and 
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19. 



circuitry for receiving in An Appropriate slot, in- 
terpretations from said grammar when a verbal 
response Is non-responsive; and 
circuitry for controlling subsequent system op- 
eration dependant upon whether saJd verbal re- 
sponse was within an expected range of non- 
response. 

The system of claim 18 wherein said subsoquont 
system operation is a return to the call How in 
progress be/ore said non-response was received. 



20. The system of claim 19 wherein said subsequent 
system operation is a switch to a now call flow. 

21. A method of establishing a call How routine, said 
metnod comprising: 

establishing a directed call flow routine by one 
or more icons, each icon representing a leg of 
said call flow, each of said legs of said directed 
call flow necessarily populated in order to com- 
plete said directed call flow routine; and 
establishing an optional call flow routine by 
icons, each icon representing a leg of safd op- 
tional call flow routines, each said leg optionally 
invoked under control of a detected phrase out- 
side of the phrases expected by said directed 
call How. 

22. The method of claim 21 further comprising: 
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26. A system of establishing a call flow routine, said 
system comprising: 

one or more icons operable for a establishing 
a directed call flow routine., each said icon rep- 
resenting a leg of said call flow, each of said 
tegs of said directed can flow necessarily pop- 
ulated in order to complete said directed can 
flow routine; and 

one or more icons operable for establishing an 
optional call flow routine, each said Icon repre- 
senting a teg of said optional call flow routines, 
each said leg optionally invoked under control 
of a detected phrase outside of the phrases ex. 
poctod by said directed call flow. 

27. The system of claim 2fi further including an antici- 
pated grammar associated with said directed call 
flow and wherein said returning to said directed call 
flow after completing an invoked optional call flow 
when said detected phrase Is from said anticipated 
grammar. 

2d. The system of daim 27 further comprising: 

control lor beginning an entirety new call flow 
when said detected phrase Is not from sard an* 
ticipsted grammar associated with said direct* 
cd call flow. 

29. The system of claim 26 further comprising: 



returning to safd directed call flow after com- 
pleting an invoked optional call flow when said 
detected phrase is from an anticipated gram- 
mar associated with said directed call flow, 

23. The method of claim 22 further comprising: 



control for changing the message of said op- 
tional call flow depending upon the context in 
as which said optional call flow was invoked. 

30. The system of claim 26 further comprising: 



control for changing the message of said op~ 
beginning an entirely new call flaw when said *o tional call flow depending upon the context in 

detected phrase is not from an anticipated which said optional call flow was invoiced and 

grammar associated with said directed call the number of times said specific optional calf 

flow. flowhasbeen Invoked within a specific session. 



24. Tho method of claim 21 further comprising: 45 



changing the message of said optional call flow 
depending upon the context in which said op- 
tional call flow was invoked. 

so 

25. Tho method of cJaim 21 further comprising: 



changing the message of said optional call flow 
depending upon the context in which said op- 
tional cat! flow was invoked and the number of ss 
times said specific optional call flow has been 
invoked within a specific session. 
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